{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "d551be10-0101-4ba6-9fb5-089614ab4836",
   "metadata": {},
   "source": [
    "# Marqo get started building your first end-to-end vector search application"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "64c9a3d0-9e26-4def-b466-a99aff82009a",
   "metadata": {},
   "source": [
    "- Docs: https://docs.marqo.ai/2.6/\n",
    "- Github: https://github.com/marqo-ai/marqo\n",
    "\n",
    "## Setup and installation\n",
    "We'll start with downloading and installing Marqo\n",
    "\n",
    "Marqo requires docker. To install docker go to Docker Docs.\n",
    "Use docker to run Marqo:\n",
    "\n",
    "\n",
    "```\n",
    "\n",
    "docker pull marqoai/marqo:latest\n",
    "docker rm -f marqo\n",
    "docker run --name marqo -it -p 8882:8882 marqoai/marqo:latest\n",
    "```\n",
    "\n",
    "Start indexing and searching! \n",
    "\n",
    "\n",
    "###########################################################\n",
    "###########################################################\n",
    "###### STARTING DOWNLOAD OF MARQO ARTEFACTS################\n",
    "###########################################################\n",
    "###########################################################\n",
    "\n",
    "\n",
    "INFO:CUDA device summary:Found devices [{'id': -1, 'name': ['cpu']}]\n",
    "INFO:SetBestAvailableDevice:Best available device set to: cpu\n",
    "loading for: model_name=hf/e5-base-v2 and properties={'name': 'intfloat/e5-base-v2', 'dimensions': 768, 'tokens': 512, 'type': 'hf', 'model_size': 0.438, 'text_query_prefix': 'query: ', 'text_chunk_prefix': 'passage: ', 'notes': ''}\n",
    "INFO:marqo.tensor_search.index_meta_cache:Last index cache refresh at 1716972650.5254366\n",
    "config.json: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 650/650 [00:00<00:00, 245kB/s]\n",
    "model.safetensors:  24%|█████████████████████████████▍                                                                                             | 105M/438M [00:55<02:57, 1.88MB/s]INFO:marqo.tensor_search.index_meta_cache:Last index cache refresh at 1716972710.9341242\n",
    "model.safetensors:  48%|██████████████████████████████████████████████████████████▉                                                                | 210M/438M [01:51<02:01, 1.88MB/s]INFO:marqo.tensor_search.index_meta_cache:Last index cache refresh at 1716972771.2433136\n",
    "model.safetensors:  74%|███████████████████████████████████████████████████████████████████████████████████████████▎                               | 325M/438M [02:53<01:00, 1.88MB/s]INFO:marqo.tensor_search.index_meta_cache:Last index cache refresh at 1716972831.537597\n",
    "model.safetensors: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 438M/438M [03:53<00:00, 1.88MB/s]\n",
    "tokenizer_config.json: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 314/314 [00:00<00:00, 18.8kB/s]\n",
    "vocab.txt: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 232k/232k [00:00<00:00, 549kB/s]\n",
    "tokenizer.json: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 711k/711k [00:00<00:00, 897kB/s]\n",
    "INFO:marqo.tensor_search.index_meta_cache:Last index cache refresh at 1716972891.8055663\n",
    "special_tokens_map.json: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 125/125 [00:00<00:00, 354kB/s]\n",
    "INFO:marqo.s2_inference.s2_inference:loaded hf/e5-base-v2 on device cpu with normalization=True at time=2024-05-29 08:50:50.428440.\n",
    "INFO:ModelsForStartup:hf/e5-base-v2 cpu run succesfully!\n",
    "loading for: model_name=open_clip/ViT-B-32/laion2b_s34b_b79k and properties={'name': 'open_clip/ViT-B-32/laion2b_s34b_b79k', 'dimensions': 512, 'note': 'open_clip models', 'type': 'open_clip', 'pretrained': 'laion2b_s34b_b79k'}\n",
    "open_clip_pytorch_model.bin:  17%|███████████████████▌                                                                                             | 105M/605M [00:55<04:26, 1.88MB/s]INFO:marqo.tensor_search.index_meta_cache:Last index cache refresh at 1716972952.1659443\n",
    "open_clip_pytorch_model.bin:  35%|███████████████████████████████████████▏                                                                         | 210M/605M [01:51<03:30, 1.88MB/s]INFO:marqo.tensor_search.index_meta_cache:Last index cache refresh at 1716973012.4907255\n",
    "open_clip_pytorch_model.bin:  49%|██████████████████████████████████████████████████████▊                                                          | 294M/605M [02:36<02:45, 1.88MB/s]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f20c3604-4e40-431a-8409-1d27e7f9acf3",
   "metadata": {},
   "source": [
    "## 测试1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "2a2abfd1-f595-4e97-a578-85c04aa888a3",
   "metadata": {},
   "outputs": [],
   "source": [
    "import marqo\n",
    "\n",
    "mq = marqo.Client(url=\"http://localhost:8882\")\n",
    "\n",
    "mq.create_index(\"my-first-index\", model=\"hf/e5-base-v2\")\n",
    "\n",
    "mq.index(\"my-first-index\").add_documents(\n",
    "    [\n",
    "        {\n",
    "            \"Title\": \"The Travels of Marco Polo\",\n",
    "            \"Description\": \"A 13th-century travelogue describing Polo's travels\",\n",
    "        },\n",
    "        {\n",
    "            \"Title\": \"Extravehicular Mobility Unit (EMU)\",\n",
    "            \"Description\": \"The EMU is a spacesuit that provides environmental protection, \"\n",
    "            \"mobility, life support, and communications for astronauts\",\n",
    "            \"_id\": \"article_591\",\n",
    "        },\n",
    "    ],\n",
    "    tensor_fields=[\"Description\"],\n",
    ")\n",
    "\n",
    "results = mq.index(\"my-first-index\").search(\n",
    "    q=\"What is the best outfit to wear on the moon?\"\n",
    ")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "4b28295d-89fe-4315-990e-c85f73f64a9c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'hits': [{'Description': 'The EMU is a spacesuit that provides environmental '\n",
      "                          'protection, mobility, life support, and '\n",
      "                          'communications for astronauts',\n",
      "           'Title': 'Extravehicular Mobility Unit (EMU)',\n",
      "           '_highlights': [{'Description': 'The EMU is a spacesuit that '\n",
      "                                           'provides environmental protection, '\n",
      "                                           'mobility, life support, and '\n",
      "                                           'communications for astronauts'}],\n",
      "           '_id': 'article_591',\n",
      "           '_score': 0.8302065112850622},\n",
      "          {'Description': \"A 13th-century travelogue describing Polo's travels\",\n",
      "           'Title': 'The Travels of Marco Polo',\n",
      "           '_highlights': [{'Description': 'A 13th-century travelogue '\n",
      "                                           \"describing Polo's travels\"}],\n",
      "           '_id': '481e7341-e8c0-4ab8-b214-a832fac7414f',\n",
      "           '_score': 0.7665057498844796}],\n",
      " 'limit': 10,\n",
      " 'offset': 0,\n",
      " 'processingTimeMs': 301,\n",
      " 'query': 'What is the best outfit to wear on the moon?'}\n"
     ]
    }
   ],
   "source": [
    "import pprint\n",
    "pprint.pprint(results)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f2bcd942-7592-4262-a00b-02ea0b135e6c",
   "metadata": {},
   "source": [
    "## 测试中文"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "2b21b813-aa95-45a7-b2e4-cb2643c6f77f",
   "metadata": {},
   "outputs": [],
   "source": [
    "mq.index(\"my-first-index\").add_documents(\n",
    "    [\n",
    "        {\n",
    "            \"Title\": \"什么是数字孪生\",\n",
    "            \"Description\": \"数字孪生是完成的物理资产、流程和系统的虚拟表示，可在整个产品生命周期中使用实时或其他来源的数据，以提供动态洞察力\",\n",
    "        },\n",
    "        {\n",
    "            \"Title\": \"what is digital thread\",\n",
    "            \"Description\": \"Digital thread is a mechanism for correlating information across multiple dimensions of the virtual representation, where the dimensions include (but are not limited to) time or lifecycle stage (including design intent), kind-of-model, and configuration history.\",\n",
    "            \"_id\": \"article_011\",\n",
    "        },\n",
    "    ],\n",
    "    tensor_fields=[\"Description\"],\n",
    ")\n",
    "\n",
    "results = mq.index(\"my-first-index\").search(\n",
    "    q=\"什么是数字主线?\"\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "5bde42c8-52c9-456f-8fdc-6c851213ac0e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'hits': [{'Description': '数字孪生是完成的物理资产、流程和系统的虚拟表示，可在整个产品生命周期中使用实时或其他来源的数据，以提供动态洞察力',\n",
      "           'Title': '什么是数字孪生',\n",
      "           '_highlights': [{'Description': '数字孪生是完成的物理资产、流程和系统的虚拟表示，可在整个产品生命周期中使用实时或其他来源的数据，以提供动态洞察力'}],\n",
      "           '_id': 'cd742485-d1a3-4c73-b265-ca15c39f18ec',\n",
      "           '_score': 0.8324224533975645},\n",
      "          {'Description': \"A 13th-century travelogue describing Polo's travels\",\n",
      "           'Title': 'The Travels of Marco Polo',\n",
      "           '_highlights': [{'Description': 'A 13th-century travelogue '\n",
      "                                           \"describing Polo's travels\"}],\n",
      "           '_id': '481e7341-e8c0-4ab8-b214-a832fac7414f',\n",
      "           '_score': 0.7797991056384933},\n",
      "          {'Description': 'Digital thread is a mechanism for correlating '\n",
      "                          'information across multiple dimensions of the '\n",
      "                          'virtual representation, where the dimensions '\n",
      "                          'include (but are not limited to) time or lifecycle '\n",
      "                          'stage (including design intent), kind-of-model, and '\n",
      "                          'configuration history.',\n",
      "           'Title': 'what is digital thread',\n",
      "           '_highlights': [{'Description': 'Digital thread is a mechanism for '\n",
      "                                           'correlating information across '\n",
      "                                           'multiple dimensions of the virtual '\n",
      "                                           'representation, where the '\n",
      "                                           'dimensions include (but are not '\n",
      "                                           'limited to) time or lifecycle '\n",
      "                                           'stage (including design intent), '\n",
      "                                           'kind-of-model, and configuration '\n",
      "                                           'history.'}],\n",
      "           '_id': 'article_011',\n",
      "           '_score': 0.7780799715206977},\n",
      "          {'Description': 'The EMU is a spacesuit that provides environmental '\n",
      "                          'protection, mobility, life support, and '\n",
      "                          'communications for astronauts',\n",
      "           'Title': 'Extravehicular Mobility Unit (EMU)',\n",
      "           '_highlights': [{'Description': 'The EMU is a spacesuit that '\n",
      "                                           'provides environmental protection, '\n",
      "                                           'mobility, life support, and '\n",
      "                                           'communications for astronauts'}],\n",
      "           '_id': 'article_591',\n",
      "           '_score': 0.7748240492489379}],\n",
      " 'limit': 10,\n",
      " 'offset': 0,\n",
      " 'processingTimeMs': 192,\n",
      " 'query': '什么是数字主线?'}\n"
     ]
    }
   ],
   "source": [
    "import pprint\n",
    "pprint.pprint(results)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1fd81889-8af3-4b7d-b23d-89b26ff9f946",
   "metadata": {},
   "source": [
    "## 效果不好的原因\n",
    "\n",
    "中英文混合不佳，解决方法：中文的要用中文，英文嵌入的要用英文"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "c9329d0a-9251-42b3-8550-a68353fb3cf2",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'hits': [{'Description': 'Digital thread is a mechanism for correlating '\n",
      "                          'information across multiple dimensions of the '\n",
      "                          'virtual representation, where the dimensions '\n",
      "                          'include (but are not limited to) time or lifecycle '\n",
      "                          'stage (including design intent), kind-of-model, and '\n",
      "                          'configuration history.',\n",
      "           'Title': 'what is digital thread',\n",
      "           '_highlights': [{'Description': 'Digital thread is a mechanism for '\n",
      "                                           'correlating information across '\n",
      "                                           'multiple dimensions of the virtual '\n",
      "                                           'representation, where the '\n",
      "                                           'dimensions include (but are not '\n",
      "                                           'limited to) time or lifecycle '\n",
      "                                           'stage (including design intent), '\n",
      "                                           'kind-of-model, and configuration '\n",
      "                                           'history.'}],\n",
      "           '_id': 'article_011',\n",
      "           '_score': 0.9302977674194587},\n",
      "          {'Description': '数字孪生是完成的物理资产、流程和系统的虚拟表示，可在整个产品生命周期中使用实时或其他来源的数据，以提供动态洞察力',\n",
      "           'Title': '什么是数字孪生',\n",
      "           '_highlights': [{'Description': '数字孪生是完成的物理资产、流程和系统的虚拟表示，可在整个产品生命周期中使用实时或其他来源的数据，以提供动态洞察力'}],\n",
      "           '_id': 'cd742485-d1a3-4c73-b265-ca15c39f18ec',\n",
      "           '_score': 0.7804216194798395},\n",
      "          {'Description': 'The EMU is a spacesuit that provides environmental '\n",
      "                          'protection, mobility, life support, and '\n",
      "                          'communications for astronauts',\n",
      "           'Title': 'Extravehicular Mobility Unit (EMU)',\n",
      "           '_highlights': [{'Description': 'The EMU is a spacesuit that '\n",
      "                                           'provides environmental protection, '\n",
      "                                           'mobility, life support, and '\n",
      "                                           'communications for astronauts'}],\n",
      "           '_id': 'article_591',\n",
      "           '_score': 0.7746530255243431},\n",
      "          {'Description': \"A 13th-century travelogue describing Polo's travels\",\n",
      "           'Title': 'The Travels of Marco Polo',\n",
      "           '_highlights': [{'Description': 'A 13th-century travelogue '\n",
      "                                           \"describing Polo's travels\"}],\n",
      "           '_id': '481e7341-e8c0-4ab8-b214-a832fac7414f',\n",
      "           '_score': 0.7474890136970018}],\n",
      " 'limit': 10,\n",
      " 'offset': 0,\n",
      " 'processingTimeMs': 298,\n",
      " 'query': 'what is digital thread?'}\n"
     ]
    }
   ],
   "source": [
    "results = mq.index(\"my-first-index\").search(\n",
    "    q=\"what is digital thread?\"\n",
    ")\n",
    "\n",
    "# import pprint\n",
    "pprint.pprint(results)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "0366a455-6d9c-4cb4-b14c-8aef77689510",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'hits': [{'Description': '数字孪生是完成的物理资产、流程和系统的虚拟表示，可在整个产品生命周期中使用实时或其他来源的数据，以提供动态洞察力',\n",
      "           'Title': '什么是数字孪生',\n",
      "           '_highlights': [{'Description': '数字孪生是完成的物理资产、流程和系统的虚拟表示，可在整个产品生命周期中使用实时或其他来源的数据，以提供动态洞察力'}],\n",
      "           '_id': 'cd742485-d1a3-4c73-b265-ca15c39f18ec',\n",
      "           '_score': 0.8780940686040424},\n",
      "          {'Description': 'The EMU is a spacesuit that provides environmental '\n",
      "                          'protection, mobility, life support, and '\n",
      "                          'communications for astronauts',\n",
      "           'Title': 'Extravehicular Mobility Unit (EMU)',\n",
      "           '_highlights': [{'Description': 'The EMU is a spacesuit that '\n",
      "                                           'provides environmental protection, '\n",
      "                                           'mobility, life support, and '\n",
      "                                           'communications for astronauts'}],\n",
      "           '_id': 'article_591',\n",
      "           '_score': 0.786959454207839},\n",
      "          {'Description': 'Digital thread is a mechanism for correlating '\n",
      "                          'information across multiple dimensions of the '\n",
      "                          'virtual representation, where the dimensions '\n",
      "                          'include (but are not limited to) time or lifecycle '\n",
      "                          'stage (including design intent), kind-of-model, and '\n",
      "                          'configuration history.',\n",
      "           'Title': 'what is digital thread',\n",
      "           '_highlights': [{'Description': 'Digital thread is a mechanism for '\n",
      "                                           'correlating information across '\n",
      "                                           'multiple dimensions of the virtual '\n",
      "                                           'representation, where the '\n",
      "                                           'dimensions include (but are not '\n",
      "                                           'limited to) time or lifecycle '\n",
      "                                           'stage (including design intent), '\n",
      "                                           'kind-of-model, and configuration '\n",
      "                                           'history.'}],\n",
      "           '_id': 'article_011',\n",
      "           '_score': 0.7808037890632545},\n",
      "          {'Description': \"A 13th-century travelogue describing Polo's travels\",\n",
      "           'Title': 'The Travels of Marco Polo',\n",
      "           '_highlights': [{'Description': 'A 13th-century travelogue '\n",
      "                                           \"describing Polo's travels\"}],\n",
      "           '_id': '481e7341-e8c0-4ab8-b214-a832fac7414f',\n",
      "           '_score': 0.7805005914739547}],\n",
      " 'limit': 10,\n",
      " 'offset': 0,\n",
      " 'processingTimeMs': 247,\n",
      " 'query': '什么是数字孪生?'}\n"
     ]
    }
   ],
   "source": [
    "results = mq.index(\"my-first-index\").search(\n",
    "    q=\"什么是数字孪生?\"\n",
    ")\n",
    "\n",
    "# import pprint\n",
    "pprint.pprint(results)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "438f6cb8-75ba-4910-958f-14aaa8e0d65e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'errors': False,\n",
       " 'processingTimeMs': 1258.0065840002135,\n",
       " 'index_name': 'my-first-index',\n",
       " 'items': [{'status': 200, '_id': '5dc24f3f-e4bd-47be-82c3-7c45751a6e4d'},\n",
       "  {'status': 200, '_id': 'article_021'}]}"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mq.index(\"my-first-index\").add_documents(\n",
    "    [\n",
    "        {\n",
    "            \"Title\": \"洛马数字孪生5级成熟度\",\n",
    "            \"Description\": \"1级：虚拟数字孪生 \\n 2级：数字孪生与实体同步 \\n 3级：数字孪生与实体相互验证 \\n 4级：数字孪生与实体集成 \\n 5级：数字孪生运行生态系统\",\n",
    "        },\n",
    "        {\n",
    "            \"Title\": \"基于仿真软件的数字孪生\",\n",
    "            \"Description\": \"仿真软件本来是用于对实际设备进行仿真。现在数字孪生概念火起来之后，仿真界突然发现数字孪生这个框用仿真软件去套简直完美。这类软件的数字孪生的实质是运行设备的仿真模型，然后通过物联网采集的数据喂给仿真模型，让模型进行实时仿真，这样就可以实现故障诊断、性能预测、控制优化等功能。.\",\n",
    "            \"_id\": \"article_021\",\n",
    "        },\n",
    "    ],\n",
    "    tensor_fields=[\"Description\"],\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "946cd519-3798-4233-a1c4-2ee9afd39ad6",
   "metadata": {},
   "outputs": [],
   "source": [
    "results = mq.index(\"my-first-index\").search(\n",
    "    q=\"基于仿真软件的数字孪生是什么?\"\n",
    ")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "b4c0c156-7de3-4ece-b157-57ca9ef8e2d0",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "'仿真软件本来是用于对实际设备进行仿真。现在数字孪生概念火起来之后，仿真界突然发现数字孪生这个框用仿真软件去套简直完美。这类软件的数字孪生的实质是运行设备的仿真模型，然后通过物联网采集的数据喂给仿真模型，让模型进行实时仿真，这样就可以实现故障诊断、性能预测、控制优化等功能。.'\n"
     ]
    }
   ],
   "source": [
    "# import pprint\n",
    "pprint.pprint(results['hits'][0]['Description'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6d01fc3e-87f5-4b67-99e4-0a4bfb4029c5",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "73b3b09b-8dc6-4961-a580-219485e6fec3",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
